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STUDIES  OF  LEARNING  AND  SELF-CONTAINED 
EDUCATIONAL  SYSTEMS,  1973-1976 


Work  on  this  contract  has  been  concerned  with  studies  of  the  teaching 
and  learning  process.  The  emphasis  has  been  on  the  nature  of  the  tutorial 
interaction  during  the  learning  of  complex  material,  with  the  eventual 
goal  of  being  able  to  construct  an  automated,  computer-controlled  tutorial 
system. 

In  order  to  understand  tutorial  instruction,  one  must  understand  how 
a person  acquires  new  knowledge.  Accordingly,  we  must  come  to  understand 
the  ways  by  which  knowledge  is  represented  within  human  memory,  how 
new  information  is  incorporated  within  the  old  knowledge  structures,  and 
how  a person  comes  to  use  the  new  information  in  appropriate  ways.  The 
tutor  must  have  knowledge  of  both  the  subject  matter  and  the  changing 
state  of  the  student.  If  we  intend  to  automate  the  instructional  process, 
we  need  to  be  able  to  understand  how  a good  tutor  operates,  and  this 
requires  us  to  understand  and  be  able  to  state: 

1.  The  representation  of  the  subject  matter  knowledge  within 
the  human  memory  system. 

2.  A model  of  the  student,  which  includes  representing  the 
knowledge  and  the  learning  process. 

3.  The  principles  and  strategies  of  instruct'd  ■ 


The  Learning  of  Complex  Material 


How  does  someone  learn  a complex  topic  matter?  Consider,  for  instance: 

1)  Learning  the  history  of  the  American  Civil  War 

2)  Learning  how  to  program  a computer 

3)  Learning  the  psychology  of  hearing 

These  topics  differ  in  style  and  form,  but  all  have  the  common  properties 
of  any  complex  topic  matter — each  topic  requires  the  integration  of  a vast 
amount  of  material.  The  most  difficult  part  of  the  learning  of  each  of 
these  topics  is  in  understanding  the  interrelationships  of  the  components 
that  make  up  each  topic.  The  factual  learning,  or  committing  to  memory  of 
the  individual  components,  is  not  any  particular  problem.  The  level  of 
performance  of  students  on  tests  of  these  topics  is  not  much  affected  by 
allowing  them  free  access  to  reference  books  during  testing.  The  complexity 
clearly  resides  in  the  interrelationships  rather  than  in  the  acquisition  of 
the  individual  components. 

The  study  of  complex  learning  poses  two  types  of  problems.  One  is 
concerned  with  the  psychological  basis  of  knowledge,  theoretical  understanding 
of  the  learning  process  and  of  the  knowledge  structure  within  human  memory. 

The  other  is  concerned  with  the  actual  presentation  of  material  to  a 

learner  and  with  assessing  the  learner’s  evolving  understanding  of  the  material. 

Learning  and  Teaching  Complex  Material 

One  feature  of  the  learning  and  teaching  of  a complex  body  of  material 
is  that  the  interrelationships  are  so  rich,  there  is  no  single  sequence  of 
presentation  that  is  adequate.  To  understand  a given  concept  fully,  one 
must  understand  its  relationships  to  all  other  relevant  concept j.  But  this 
means  understanding  those  other  concepts  as  well,  and  they  in  turn  require 
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a knowledge  of  the  concepts  with  which  they  interrelate.  Therefore,  complex 
material  must  be  presented  in  a way  t.iat  allows  the  student  f develop  some 
framework  for  relating  the  pieces  of  information  to  each  other,  and  then 
covers  the  material  again,  perhaps  in  more  depth,  enlarging  and  enriching 
the  framework  of  knowledge  already  acquired.  The  effective  teacher  or 
textbook  author  follows  this  approach,  not  necessarily  being  aware  of  the 
rules  being  followed.  We  have  called  this  approach  web  teaching  (Norman, 
1973). 

An  observation  of  the  process  of  learning  complex  topics  suggests 
some  important  properties  that  any  theoretical  description  must  be  capable 
of  handling.  The  learning  of  complex  topics  has  continuity,  with  the  new 
knowledge  being  added  to  previously  acquired  knowledge.  The  knowledge  can 
exist  at  many  levels  of  specification.  It  is  possible  to  grasp  a topic 
briefly  and  quickly,  or  to  elaborate  upon  it  indefinitely.  This  implies 
that  knowledge  about  a topic  is  incremental.  New  concepts  can  be  added  to  the 
previous  knowledge  base  almost  indefinitely.  Knowledge  of  complex  topics  is 
robust.  This  does  not  imply  that  things  are  never  forgotten  or  that  mistakes 
are  not  made,  but  rather  that  a basic  structure  for  the  material  can  survive 
even  large  errors  or  large  gaps  of  knowledge.  The  robustness  of  complex 
knowledge  seems  to  haye  two  causes:  1)  any  given  unit  of  knowledge  is 

determined  by  multiple  sources;  2)  it  is  possible  to  derive  many  of  the  units 
from  other  things  that  are  known.  Thus,  complex  topics  are  indeed  complex. 
They  consist  of  an  interlocking  network  of  concepts,  each  of  which  contributes 
to  the  overall  knowledge,  but  none  of  which  is  indispensible . 
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Studies  of  the  Representation  of  Knowledge 

To  understand  what  happens  when  a person  requires  knowledge  of  a 
complex  topic,  one  must  know  of  the  operations  of  the  human  memory 
system,  how  material  is  entered  into  that  sastem,  how  it  is  represented, 
and  how  it  is  later  retrieved  and  used.  These  are  complicated  issues 
which  are  at  the  c irrent  state  of  the  art  in  psychology.  New  developments 
in  our  understanding  of  memory  structures  have  occurred  in  recent  years, 
and  it  appears  that  we  are  discovering  the  basic  structures  of  human 
memory,  of  the  knowledge  within  i',  and  therefore,  of  the  process  of 
learning.  Considerable  progress  has  been  made  on  this  problem  within 
the  contract  research  period. 

Semantic  networks.  A number  of  workers  in  the  field  of  human  memory 
have  recently  been  developing  formal  representations  for  thi  knowledge  within 
memory.  Most  of  the  models  that  we  are  interested  in  here  are  described  as 
semantic  networks  and,  following  the  work  of  Quillian  (1968),  are  all 
characterized  by  a directed,  labelled  graph  structure.  Carbonell  (1S'70) 
used  semantic  networks  for  the  knowledge  base  in  a computer-assisted 
instructional  system,  and  the  work  has  continued  under  the  direction  of 
Collins.  The  version  we  have  helped  develop  is  called  an  active  structural 
network  to  emphasize  that  the  representation  is  both  active  and  passive,  and 
that  it  can  contain  general  procedures  that  can  be  executed  whenever 
functional  knowledge  must  be  used.  (This  work  is  reported  in  Norman,  Rumelhart 
and  the  LNR  Research  Croup,  1375;  we  will  refer  to  it  as  the  LNK  work.) 
Although  we  base  our  discussion  on  the  LNR  work,  in  fact  the  comments  apply 
to  al3  semantic  network  representations. 

Basically,  the  semantic  network  provides  a means  of  representing 
knowledge.  It  is  a new  tool  in  psychology.  Previously,  our  formal  models 
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have  been  abstract.  For  example,  a mathematical  learning  model  predicts 
the  probability  of  the  strength  of  some  association  between  two  elements, 
but  the  elements  are  usually  part  of  a large  homogeneous  set.  With  semantic 
networks,  we  100k  at  the  structure  of  very  particular  items. 

The  internal  components  of  verbs.  Consider  how  a child  might  come  to 
learn  language.  In  the  work  of  the  LNR  Research  Group,  verbs  can  be  decomposed 
into  underlying  primitive  elements.  Thus,  some  verbs  specify  only  STATIVE 
components;  others  are  more  complex,  specifying  CAUSE  and  CHANGE.  The 

statement  that 

(1)  The  skier  went  to  the  top  of  the  mountain, 
is  represented  as  shown  in  Figure  1,  which  shows  the  decomposition  of  the 
verb  rwent"  into  its  more  basic  underlying  components. 


Figtre  1.  The  skier  went  to  the  top  of  .the  mountain. 


Network  o kl,. "tures  are  composed  of  nodes  and  ordered,  labelled  relations 


connecting  these  nodes.  In  figure  1,  the  ovals  represent  nodes  that  are 
token  instances  of  propositional  structures,  the  angular  brackets  (e.g., 

< skier >)  represent  nodes  that  are  token  instances  of  concepts,  and  the  phrases 
in  quotation  marks  (e.g.,  "bottom  of  mountain")  represent  network  structures 
for  the  concepts  described  by  the  phrases,  but  which  we  have  not  shown  in 
detail  in  the  figure  in  order  to  preserve  the  diagram's  clarity.  (1*ie  numbers 
of  the  token  propositional  nodes  have  no  intrinsic  meaning:  they  are  used  to 
facilitate  reference  to  them.  Node  *1  is  called  "node  star  1,"  for  example.) 

Figure  1 can  be  interpreted  in  a straightforward  manner  by  starting 
with  node  *1,  the  oval  labelled  CHANGE.  This  indicates  that  some  change 
of  state  has  taken  place:  the  relations  leaving  node  *1  describe  the  states 
that  are  involved.  The  CHANGE  takes  place  from  a state  shown  by  node  *2, 
which  says  that  its  subject  (a  skier)  was  located  at  the  "bottom  of  mountain" 
from  some  unknown  time  to  some  time  not  specified  (but  indicated  by  the 
unnamed  node  shown  as  angular  brackets) . The  result  of  the  change  is  the 
state  represented  by  node  *3:  the  skier  is  now  at  location  "top  of  mountain." 
Notice  that  the  time  he  was  located  at  the  top  of  the  mountain  is  not 
specified,  except  to  indicate  that  it  is  later  than  the  time  at  which  he  was 
no  longer  at  the  bottom  of  the  mountain. 

A more  complex  verb  is  one  that  implies  causality:  for  example,  "give." 
The  statement 

(2)  Suzette  took  the  skis  from  Henry, 
means  that  Suzette  did  something  that  caused  the  skis  to  change  from  the 
possession  of  Henry  to  herself.  The  structure  for  sentence  2 is  shown  in 
Figure  2 . 
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Figure  2.  Suzette  rook 
the  skis  from  Henry. 


Note  that  these  structures  provide  frameworks  for  adding  later  knowledge. 
Each  structure  is  a memory  schema:  it  allows  us  to  organize  material  that  is 
learned  later  on.  The  statement 

(3)  She  promised  to  return  them  by  morning. 
ad4s  information. to  our  previous  picture.  Now.we.knpw  that  Suzette  has  an 


obligation  to  return  the  skis.  We  can  deduce  that  she  got  the  skis  from 
Henry  with  his  permission,  and  so  we  can  expand  the  framework  for  the 
knowledge  to  something  like  that  shown  in  Figure  3. 


Figure  3.  Within  dotted  line:  Suzette 
took  the  skis  from  Henry.  Outside 
dotted  line:  She  promised  to  return 
them  by  tomorrow.  Total  structure: 
Suzette  borrowed  the  skis  from  Henry 
promising  to  return  them  the  next  day 


• Wi'V'.  • ‘V-.  ? 


7 


Note  that  we  add  the  new  knowledge  directly  to  the  framework  for  the 
old.  Thus,  not  only  did  we  have  a convenient  way  of  modifying  the  structure 
for  the  previous  episode  according  to  the  framework  provided  by  the  schema 
for  the  first  sentence,  but  we  have  now  modified  the  structure  into  one 
that  is  equivalent  to  the  concept  of  "borrow":  Suzette  borrowed  the  skis. 

It  is  easy  to  see  how  similar  statements  of  added  information  could  have 
modified  the  structure  to  indicate  that: 

(4)  Suzette  stole  the  skis  (no  permission  was  granted). 

(5)  Suzette  purchased  or  rented  the  skis  (she  paid  money  for  tham) . 

(6)  Suzette  got  the  skis  by  asking  Peter  to  pick  them  up  for  her 
(expanding  the  DO  statement) . 

The  representation  of  simple  sentences  such  as  these  is  only  a very 
small  step  toward  the  representation  of  complex  topic  matters.  But  it 
provides  a start. 

To  answer  a question  intelligently  requires  a large  body  of  specific 
knowledge  about  the  area  being  questioned,  as  well  as  general  knowledge 
about  the  world  and  its  causal  and  physical  laws,  and  also  an  understanding 
of  the  knowledge  and  behavior  of  other  people.  Those  of  us  trying  to  model 
the  human  use  of  memory  have  just  barely  grown  used  to  the  fact  that  we 
must  add  knowledge  of  the  world  to  the  model,  so  it  comes  as  a surprising 
and  new  challenge  that  we  must  also  incorporate  a person's  understanding 
of  other  people  into  the  model. 

A parallel  development  is  occurring  in  linguistics.  The  linguist  is 
realizing  that  real  world  •">wledge  and  an  interpretive  ability  which  ties 
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general  social  and  linguistic  knowledge  to  specific  situations  must  be 
in,  uied  in  speaker-hearer  models  to  account  for  tne  language  user  s 
ability  to  produce  and  understand  sentences  (Mehan  (1972).  A model  which 
only  incorporates  formal  grammatical  considerations  simply  cannot  account 
for  the  complexities  of  language  usage  in  interactional  situations. 


The  Acquisition  of  Knowledge 


We  have  exam:  ,ied  a number  of  different  features  about  the  representation 
of  information  in  memory.  An  active  representation  seems  to  be  necessary, 
with  previously  acquired  knowledge  about  the  world  providing  a structural 
framework  upon  which  to  construct  new  knowledge.  The  pre-existing  structure 
not  only  defines  the  way  that  new  information  will  be  represented,  but  it 
helps  in  organizing  the  information  that  is  not  yet  known.  Thus,  the 
structure  can  maintain  open  positions  within  its  framework  for  the  necessary 
causes  and  results,  jbjects  and  actors,  and  even  methods  and  actions  that 
may  remain  unspecified.  In  this  way,  students  acquiring  knowledge  can  be 
led  to  seek  more  information  to  fill  the  missing  nodes  in  their  developing 
memory  representations . 


Learning 

Teaching  and  learning  are  the  names  that  we  give  to  the  activities 
of  presenting  and  acquiring  new  information.  There  is  no  adequate  theory 
of  instruction  to  guide  us  in  these  activities,  but  an  examination  of  the 
structure  of  memory  from  the  viewpoint  represented  here  appears  to  provide 
useful  guidelines  for  the  development  of  such  a theory.  Indeed,  MacDonald- 
Ross  (1972)  has  studied  network  relationships  with  just  this  goal  in  mind, 
to  determine  how  to  design  an  educational  system  for  the  Open  Univei  sity  in 
England.  MacDonald-Ross  argues  that  structures  of  this  nature  do  guide 
both  students  and  faculty  into  an  understanding  of  their  subject  matter. 
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Examine  the  structure  of 


memory.  It  is  an  interconnected  network, 
with  new  facts  supported  within  the  skeletal  structure  provide  by  the  old. 

To  acquire  new  information  means  to  construct  new  nodes  and  the  relations 
between  them.  Consider  how  that  might  be  accomplished.  First,  let  Figure  4A 
represent  a segment  of  knowledge  in  the  memory.  If  we  acquire  two  new 
concepts  (C1,  C2)  and  a relation  between  them  (R)  as  shown  in  Figure  4B, 
there  are  no  connections  to  the  old  network.  Retrieval  should  be  difficult, 

and  perhaps  even  the  acquisition  is  difficult.  This  is  poorly  learned 
information. 

Now  consider  Figure  4C.  Here,  the  two  newly  acquired  nodes  are  well 
integrated.  To  use  an  obvious  analogy  drawn  from  the  structure  itself,  we 
would  say  that  the  newly  learned  components  of  4C  are  well  supported;  those 
of  4B  lack  support.  Notice,  too,  that  4C  is  better  integrated  within  the 
network,  in  part  because  it  contains  more  information.  If  this  view  of 
things  is  correct,  it  explains  why  mnemonic  techniques  are  so  useful  in 
causing  arbitrary  strings  of  items  to  be  learned  even  though  they  add  to 
the  total  amount  of  material  that  is  to  be  acquired.  They  provide  a firm, 
well  integrated  structural  framework. 


Figure  4.  A network  representation  of  the  knowledge  structure  in  memory 


Teaching 


Suppose  we  have  a large  body  of  knowledge  to  teach — what  would  be 
the  best  way  of  doing  it?  Presumably,  we  need  to  interconnect  the  new 
information  with  the  existing  structure.  One  way  to  do  this  is  to  construct 
a supoorting  web  structure  first,  and  then  fill  in  the  details.  ”0  do 
the  details  first  would  not  work,  for  without  a supportirg  structure,  the 
new  material  simply  could  not  become  integrated.  In  teaching,  this  means 
thr t an  outline  of  ^he  material  to  be  learned  should  be  acquired  first, 
then  a more  detailed  overview,  and  then  progressively  more  and  more  detailed 
structures . 

The  network  representation  of  knowledge  can  guide  the  process  of 

instruction  in  two  different  ways:  1)  if  we  have  a good  representation  of 

the  knowledge  we  wish  to  teach,  then  we  can  organize  it  properly  for  efficient 

learning;  2)  if  we  try  to  discover  the  network  representation  of  the  student, 

we  can  use  this  to  guide  our  teaching.  Knowing  the  knowledge  structure  of 

the  student  helps  in  devising  the  original  level  of  organization  of  the 

material.  In  addition,  as  the  lessons  progress,  we  can  use  our  understanding 

of  the  student's  developing  structures  to  guide  us  in  teaching,  telling  us 

what  old  material  has  uot  been  acquired  and  what  new  material  might  perhaps 

already  be  known.  Thus,  it  is  theoretically  possible  to  tailor  instruction 
to  the  knowledge  base  and  competence  of  the  student. 

Whether  the  network  representation  makes  the  goal  easier  to  attain 
remains  to  be  seen.  The  major  drawbacks  have  resulted  from  the  expense  of 
using  tutorial  methods  in  mass  education,  and  from  the  lack  of  sophistication 
in  the  implementation  of  most  teaching  machine  programs  and  computer-assisted 
instructional  systems  to  attempt  these  goals  (with  the  exception  of  the 


demonstration  systems  of  Brown,  Burton  & Zdybel,  1572;  Carbonell,  1970) . 
Nonetheless,  the  analysis  is  instructive  even  if  the  complete  implementation 
remains  in  the  future. 

We  can  characterize  two  different  strategies  of  presenting  material:  two 
different  strategies  of  teaching.  One  is  to  present  a cohesive  organized 
structure  to  the  student,  carefully  adding  one  new  piece  of  information  after 
another  to  the  developing  structure.  This  might  be  called  linear  teaching. 

It  is  the  system  the  characterizes  lectures,  textbooks,  and  even  the  structure 
of  this  report.  The  other  method  is  to  present  a coarse  web  of  information, 
outlining  the  topics  to  be  discussed,  then  giving  a general  overview  followed 
by  more  detailed  overviews,  and  finally  the  detailed  substructure.  This 
procedure  might  be  called  web  teaching.  Web  teaching  is  often  prescribed, 
seldom  done.  It  is  difficult  to  perform  well.  But  we  wish  to  suggest  that 
for  the  learning  of  complex  topic  matters,  web  teaching  may  at  times  be  more 
efficient. . 

The  distinction  here  between  web  and  linear  strategies  are  closely 
related  to,  yet  different  from,  the  distinction  between  holistic  and  serialistic 
strategies  of  learning  (Pask,  1971;  Pask  & Scott,  1972).  Pask  and  Scott  gave 
students  freedom  to  explore  a network  of  knowledge  about  probability  theory. 

Some  students,  the  serialists,  preferred  to  work  on  one  node  at  a time, 
always  completing  that  node  before  proceeding.  Others,  the  holists,  worked 
on  many  nodes  simultaneously.  These  are  interesting  results  that,  hopefully, 
can  be  used  to  characterize  styles  of  learning. 

Linear  teaching.  Linear  teaching  (and  its  complement,  linear  learning) 
might  be  characterized  by  the  sequence  shown  in  Figure  5.  At  first  there  is 
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a well  developed  structure  structure  of  knowledge  to  which  we  add  a new  set 
of  nodes,  say  the  linear  string  C±  and  C2  and  their  subnodes,  as  shown  in 
Figure  5A.  Further  learning  adds  another  node  and  more  subnodes  (Figure  5B) . 
Eventually  a longer  linear  string  is  acquired,  nodes  C±  through  Cb  in 
pw  gure  5C.  Presumably,  as  learning  increases  the  older  material  becomes 
more  thoroughly  embedded  into  the  original  knowledge  structure,  but  the 
overall  picture  remains  one  of  a systematic  linear  increase  in  the  knowledge 
base.  The  new  knowledge  is  weakly  supported,  ifi  one  link  in  the  chain  is 
lost,  either  Because  it  was  poorly  acquired  or  because  the  learner  might 
have  missed  expost -e  to  that  part  of  the  material,  then  the  rest  of  the 
structure  is  weakened  and  may  easily  become  irretrievable.  This  is  a familiar 
experience  to  many  instructors— a student  who  failed  to  learn  an  important 
point  made  early  in  a course  may  thereafter  be  in  difficulty,  unable  to 
understand  most  of  what  has  followed  from  it. 


Figure  5.  Linear  learning. 
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Web  teaching.  We 3 teaching  (and  web  learning)  might  be  characterized 
by  the  sequence  shown  In  Figure  6.  Here,  we  use  the  first  few  nodes 
acquired  (Figure  6A)  to  establish  a coarse  web  which  is  well  integrated 
with  the  previous  knowledge  structure.  Then  we  insert  refinements  within 
the  structure  created  by  the  original  web  framework,  as  shown  in  Figure  6B. 
Finally,  we  can  fill  in  the  details,  as  shown  in  Figure  6C.  In  web  learning, 
no  single  node  is  critical  to  the  whole,  so  that  poor  acquisition  or  even 
the  absence  of  a single  set  of  concepts  does  not  destroy  the  validity  of 
the  structure. 

The  structures  described  in  these  two  diagrams  (Figures  5 and  6)  are  at 
best  weak  analogies  to  the  processes  of  learning  and  teaching.  As  yet,  they 
are  Ear  from  being  a formal,  testable  theory.  But  the  analogies  have  led 
to  useful  insights  at  this  early  stage  of  development  in  the  planning  of 
course  material.  Whether  there  is  more  to  the  distinctions  between  web  and 
linear  teaching  than  superficial  analogies  remains  to  be  investigated. 


Figure  6.  Web  learning. 


Studies  of  Tutorial  Instruction 


In  examining  how  a person  learns  a complex  topic  matter,  we  lace  a 
jiajor  difficulty  with  the  active  structural  network  (and  with  all  work  based 
on  a semantic  network).  The  representation  as  originally  formulated  did  not 
readily  allow  for  different  types  of  knowledge.  That  is,  with  the  representa- 
tional system  it  is  possible  to  represent  individual  events  and  concepts,  but 
not  easy  to  represent  higher  level  conceptualizations,  or  causes  of  those 
events.  For  example,  it  is  easy  to  represent  that  General  Grant  smoked  cigars 
or  that  the  Amazon  River  had  numerous  tributaries,  but  not  easy  to  represent 
that  the  North's  plan  of  action  in  the  Civil  War  was  to  cut  off  the  South  from 
external  supplies,  and  that  this  plan  motivated  Lincoln  to  keep  a major 
segment  of  troops  at  Cairo,  Illinois  and  St.  Louis,  and  that  it  eventually 
led  Grant  to  cut  off  the  Tennessee  and  Cumberland  rivers.  High  level 
summaries  of  plans  and  goals  that  motivate  behavior  must  be  part  of  any 
knowledge  base  about  the  war.  At  the  time  the  contract  research  was  started, 
little  was  known  about  the  representation  of  causes,  motives,  plans,  and 
goals,  making  it  impossible  to  represent  complex  interrelations  among  the 
events  that  characterize  most  real  situations. 

Event  Schemas 

In  1974,  Rumelhart  developed  a representational  scheme  for  handling 
causal  events  within  the  framework  of  the  LNR  structural  network  (described 
in  Rumelhart,  1975).  Rumelhart ' s work  was  important  in  revealing  the 
underlying  struccure  of  complex  episodes,  and  we  were  able  to  adapt  the 
story  schema  that  he  had  originally  intended  for  use  in  the  analysis  of 
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fairy  tales  to  more  complex  material — in  particular,  to  event  schemas 
appropriate  for  representing  the  American  Civil  War. 

Many  stories  have  a readily  apparent  structure:  first  an  original 
setting  for  the  story  is  presented,  and  then  an  episode.  Within  the  episode, 
there  is  a series  of  events  that  befall  the  protagonist,  and  each  event 
initiates  some  reaction.  These  aspects  of  stories  can  be  captured  by 
simple  functions  (which  we  call  predicates)  that  relate  settings  to 
episodes,  events,  and  goals.  Hence,  a story  can  be  decomposed  into  a 
setting  that  "allows"  an  episode: 

(7)  ALLOW  (setting,  episode) 

Similarly,  an  episode  occurs  when  some  event  initiates  a reaction: 

(8)  INITIATE  (event,  reaction) 

In  addition,  we  have  causes  and  motivations  to  consider,  complete  with 

internal  responses  of  the  actors  and  their  overt  responses  as  they  react. 

An  actual  historical  event,  such  as  the  American  Civil  War,  is  more 

complex  than  stories  in  chat  there  are  numerous  protagonists,  each  often 

with  independent  and  conflicting  motivations.  The  interactions  of  these 
protagonists  are  the  causal  events  that  trigger  a good  deal  of  the  war  action. 

Sometimes  the  events  are  premeditated,  sometimes  they  result  from  the 

accidental  congruence  of  the  actors.  The  basic  ideas  of  the  story  schema 

are  compatible  with  events  such  as  the  Civil  War  even  if  the  details  are  not. 

With  the  assistance  of  Rumelhart,  we  de\ eloped  an  event  schema  representation 

of  the  Civil  War. 

The  Civil  War  representation  structures  the  aims,  goals,  and  motivations 
of  the  protagonists  of  the  war.  It  enables  us  to  represent  such  issues  as: 


(9)  Why  was  Fort  Henry  important? 

(10)  What  was  the  southern  strategy  in  1862? 

(11)  Where  was  the  southern  line  of  defense  after  the 
Battle  of  Shiloh? 

In  addition,  because  of  its  heavy  emphasis  on  ctusal  factors,  the  Civil  War 
structure  enables  the  answering  of  causal  questions  at  several  levels  of 
analysis : 

(12)  Question:  Why  did  Grant  capture  Fort  Donelson? 

Answer  1:  Because  he  bombarded  the  Fort  from  both  land  and  sea. 
Answer  2:  Because  he  wished  to  drive  the  Confederates  from  the 
Fort. 

Answer  3:  Bei  -.use  he  was  only  12  railes  away,  at  Fort  Henry. 

Answer  4:  Because  Fort  Donelson  was  on  the  Cumberland  River. 

Answer  5:  Because  the  Cumberland  River  was  an  important 
transportation  route. 

Answer  6:  Because  it  was  important  that  the  North  gain  control 
of  the  western  rivers. 

Each  of  these  different  answers  is  correct,  but  which  is  appropriate  depends 
upon  the  particular  situation.  With  the  event  schema  representation,  all  of 
these  answers  are  in  the  data  base,  and  the  only  problem  is  to  develop  the 
strategies  for  determining  which  levels  of  analysis  are  appropriate  at  any 
given  time. 
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The  Civil  War  Tutorial  System 


Consider  how  such  a knowledge  representation  might  be  used.  Figure  7 A 
shows  a small  section  of  the  data  base  that  represents  the  Civil  War 
battles  around  the  forts  on  the  Tennessee  and  Cumberland  Rivers  (Forts 
Henry  and  Donelson) . The  data  base  was  encoded  according  to  the  event 
schemas  appropriate  to  the  Civil  War.  To  create  the  data  base,  we  used 
the  MEMOD  computer  system  which  we  have  developed  (Norman,  Rumelhart,  & the 
LNR  Group,  1975),  using  the  SOL  language  parts  of  that  system  to  define 
verbs  and  other  words  that  create  the  appropriate  structures.  Then  we 
constructed  the  text  shown  in  Figure  7B  and  typed  it  into  the  system, 
which  automatically  created  the  appropriate  memory  structures. 


Figure  7.  The  data  base. 

A)  Schematic  representation  of  the  data  base. 
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B)  Text  used  to  create  the  data  base. 


The  tutorial  interaction  was  performed  with  the  aid  of  the  computer 
system  through  a technique  called  "incremental  simulation."  We  automate 
as  much  of  the  system  as  is  possible,  -md  allow  humans  to  enact  the  roles 
of  those  components  of  the  system  not  yet  automated.  This  technique  enables 
us  to  test  our  ideas  before  all  the  details  of  the  system  are  fully 
understood.  It  also  permits  us  to  make  changes  mere  rapidly  as  we  learn 
about  the  componen : Interactions  than  would  be  possible  with  full  simulation. 
Figure  8 (A  and  B)  presents  the  conceptual  models  of  the  student  and  utor. 

In  addition  to  the  data  hase.  there  is  a set  of  routines  to  help  the 
tutor  interact  with  both  the  data  base  and  the  student.  The  instructor 
and  the  student  sit  at  computer  terminals  and  communicate  through  the 
computer.  The  student  can  talk  only  to  the  instructor,  but  the  tutor 
can  talk  to  the  student  or  to  the  different  parts  of  the  system.  These 
communication  pathways  are  illustrated  in  Fir  re  8C. 


A)  MODEL  or  STUDEKT 


C)  MODEL  OF  THE  TUTORIAL  SYSTEM 


B)  MODEL  OF  TUTOR 
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Figure  8.  Diagram  of  the  Civil  War  Tutorial  System. 
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The  tutor  can  ask  questions,  evaluate  the  student's  answers,  and 
present  information  in  several  ways.  The  tutor  can  discuss  a topic  by 
finding  one  piece  of  information  in  the  data  base  and  travelling  along 
the  links  of  the  network  to  get  to  other  pieces.  This  can  be  done  one 
link  at  a time  or  by  using  a semi-automated  version  that  will  continue 
along  a causal  chain  asking  questions.  (These  network-travelling  routines 
are  an  initial  step  toward  the  development  of  automated  teaching  strategics.) 
The  tutor  may  also  supply  a piece  of  information  directly  by  typing  it  in 
rather  than  using  the  data  base. 

Currently  the  system  leaves  control  and  initiative  up  to  the  tutor. 

It  does  not  prompt  to  ask  or  present  or  evaluate,  although  an  evaluation 
will  be  required  by  the  routines  that  encode  the  model  of  the  student.  A 
sample  (simulated)  dialogue  using  the  system  is  shown  in  Table  1. 


TUTOR  TO  SfSTCM 


Table  1.  Simulated  Dialogue 

SYSTEM  TO  STUOENT 


STUOENT  TO  TUTOR 


1.  STATE  "*0ESIRE-3016". 


3.  FOLLOW  "MOTIVATE". 


2.  THE  NORTH  OESIRED  TO  STOP  TRAFFIC  ON  THE 
WESTERN  RIVERS. 

A.  WHAT  010  THE  NORTH  0ECI0E  TO  00  THEN? 


6.  RIGHT. 

8.  THE  NORTH  OECIDEO  TO  ATTACK 
FORT  HENRY  8Y  BOAT  AilO  BY  LANO. 


7.  RIGHT. 


9.  THE  NORTH  0ECI0E0  TO  ATTACK  FORT  HENRY  BY  BOAT  ANO 
BY  LANO. 

10.  WHO  MARCKE0  TO  FORT  HENRY7 


13.  WRONG. 

I A.  GRANT  M.  RCHE0  TO  FGRT  HENP\ 

16.  STATE  “*SURREN0ER-313A“. 

18.  GO  OP  A LEVEL,  THEN  GO  UP 
ANOTHER  LEVEL  ANO  ASK. 


13.  WRONG. 

15.  GRANT  MARCHED  TO  FORT  HENRY. 

17.  TILGHMAN  SURREN0ERED  FORT  HENRY. 

19.  WHAT  010  FOOTE  00? 


31.  RIGHT. 

32.  RIGHT. 

24.  STATE  "VOCATE-2989" . 

25.  FORT  HENRY  IS  ON  THF  TENNESSEE  RIVER.  FORT 
00I.ELS0N  IS  ON  THE  CUMBERLAND  RIVER. 

36.  STATE  "LOCATE-32’7". 

27.  FORT  HENRY  IS  ON  THE  TENNESSEE,  ABOUT  12  MILES 
FROM  FORT  00NELS0N,  ANO  ABOUT  100  MILES  FROM 
CAIRO,  ILLINOIS. 


5.  ATTACK  FORT  HENRY. 


11.  FOOTE. 


20.  SHELLED  FORT  HENRY  FROM 
GUN  BOATS. 


23.  ASK  TUTOR  WHERE  IS  FORT  HENRY. 


Summary  of  the  Civil  War  Tutorial  Program 

We  have  been  able  to  represent  a considerable  segment  of  the  events 
and  conditions  of  the  Civil  War  in  the  computer  data  bant,  including  the 
causal  structure  of  the  actions  during  the  war.  We  are  ab3 2 to  use  the 
data  base  to  interact  with  a student,  by  using  a mode  in  which  a human 

tutor  directs  the  system  how  to  progress  through  the  data  base.  This 
tutorial  system  has  met  a number  of  our  basic  objecti  , although  it  is 

not  automated  and  still  requires  a human  tutor. 

The  basic  premises  underlying  the  tutorial  strategy  just  described  are: 

1)  The  tutor  assesses  the  knowledge  of  the  student  and  constructs 
a model  of  the  student's  knowledge  of  the  topic  matter. 

2)  The  tutor  compares  the  knowledge  of  the  student  with  the 
topic,  marking  t^e  gaps  and  inconsistencies. 

3)  The  tutor  piesento  the  material  nev-led  to  fill  in  the  gaps 
in  the  student's  knowledge. 

4)  The  presentation  of  new  material  follows  some  basic 
organizational  principles  in  terms  of  web  teaching  strategy. 
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Studies  of  Tutorial  Interaction  in  FLOW 


We  have  examined  in  detail  the  developing  representational  structure 
of  students  learning  FLOW,  a simple  algebraic  programming  language.  This 
work  has  been  reported  in  technical  reports  (Norman,  1973;  Norman,  Gentner 
& Stevens,  1974)  and  as  a book  chapter  (Norman,  Gentner  & Stevens,  1976). 

The  most  important  issues  concern  the  representation  of  the  knowledge  cf 
the  student.  We  believe  it  best  characterized  by  means  of  memory  schemas, 
small,  self-contained  units  of  procedural  knowledge.  We  shovvd  that  when 
students  encounter  a new  concept,  such  as  the  JUMP-TO  instruction  in  FLOW, 
they  impart  some  structure  to  their  interpretation  of  that  instruction. 

When  the  structure  proves  to  be  wron*. , then  the  students  must  use  their 
knowledge  of  the  behavior  of  the  program  and  the  discrepancies  from  their 
predictions  to  modify  their  schemas.  The  process  s iterative,  with  each 
cycle  taking  the  students  closer  to  the  accurate  representation  of  the 
instruction.  In  all  of  this,  the  schema  is  the  unit  that  is  of  interest, 
representing  the  end  product  of  the  learning  sessions. 

A schema  is  a basic  unit  of  knowledge.  It  provides  a framework  upon 
which  different  elements  of  information  about  a topic  can  be  integrated. 
Presumably,  schemas  are  small  in  size,  probably  limited  by  the  amount  of 
material  that  can  be  held  within  short-term  memory.  Thus,  schemas  must 
refer  to  other  schemas  in  order  to  cover  any  reasonable  amount  of  material. 
When  schemas  refer  to  one  another,  we  have  proposed  that  they  do  so  through 
context-dependent  descriptions  (Bobrow  & Norman,  1975).  By  this  we  mean 
that  the  relations  among  schemas  are  relative  to  the  context  in  which  they 
are  used,  so  that  different  contexts  may  very  well  use  different  schemas  to 
fill  the  same  description.  This  lows  for  the  use  of  matephorical  structures. 
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The  important  lemons  ltarned  from  our  studies  of  schemas  were  that  in  teaching, 
one  must  build  upon  a framework  of  material.  Students  must  understand  the 
limitations  of  their  own  knowledge,  and  they  must  learn  to  evaluate  and 
modify  their  own  schemas. 

FLOW  is  a simple,  interactive  computer  language  whose  commands  are 
designed  for  string  manipulation.  FLOW  is  specifically  aimed  at  the 
beginning  computer  programming  student,  and  has  a number  of  advantages  as 
a subject  matter  for  use  in  studies  of  learning.  Like  many  other  common 
computer  languages,  FLOW  involves  both  conceptual  and  procedural  knowledge. 

It  is  problem  oriented.  There  is  a large  variety  of  problems  that  can  be 
posed  to  exercise  the  student's  developing  knowledge.  The  language  itself 
and  the  tasks  form  a small,  concise,  and  well  defined  body  of  knowledge. 

The  statement  of  a problem  and  what  consitutes  an  acceptable  solution  are 
normally  fairly  clear,  although  there  may  be  some  debate  about  the  quality 
of  a particular  solution. 

The  students  in  most  of  our  studies  were  university  undergraduates, 
and  it  is  relatively  easy  to  find  subjects  from  that  group  with  little  or 
no  experience  with  computer  languages.  Since  FLOW  is  quite  a simple  language, 
a lot  of  interesting  learning  takes  place  within  the  first  hour  or  two.  The 
subject  matter  is  complex  enough,  however,  to  require  up  to  10  hours  to  master. 
This  forms  a convenient  time  span  for  experimental  studies.  Subjects  usually 
enjoy  learning  FLOW  with  its  interactive  nature  and  low  level  of  frustration, 
and  are  highly  motivated  to  learn. 
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The  FLOW  System 

The  FLOW  system  is  based  around  a minicomputer  with  a CRT  display 
terminal  at  which  the  student  works.  It  is  connected  to  the  Burroughs 
6700  at  the  University  of  California,  San  Diego  Computer  Center.  This 
large  computer  contains  other  programs,  such  as  the  automated  tutor,  which 
interact  with  the  student.  There  are  facilities  foi  recording  complete 
protocols  of  experimental  sessions  including  displays  on  computer  terminals 
as  well  as  voices.  If  the  experimenter/tutor  wishes  to  work  from  another 
room,  a television  monitor  duplicates  the  display  on  the  student's  terminal, 
and  there  are  terminals  for  typed  communication  with  the  student. 

The  FLOW  Tutor 

One  of  our  projects  which  utilized  the  FLOW  system  was  the  development 
of  a computer-based,  automated  tutor  which  assists  a student  who  is  learning 
FLOW.  (The  FLOW  tutor  has  been  described  in  detail  by  Centner,  Wallen  & 
Miller,  1974.)  Our  basic  purpose  was  to  investigate  how  a tutor  follows 
a student's  progress  and  helps  when  there  is  difficulty.  The  method  we  used 
was  to  construct  a computer  simulation  of  a human  tutor  who  watches  over  the 
shoulder  of  a student  learning  FLOW  from  a printed  instruction  booklet. 

In  operation,  the  sf adent  forked  at  a computer  terminal  using  an 
instruction  booklet  containing  a series  of  programming  problems  along  with 
descriptive  text  and  practice  examples.  The  FLOW  tutor  monitored  the 
student’s  keyboard  inputs  and  continually  updated  a simple  model  of  the 
student  progressing  through  the  instructions.  The  student  model  consisted 
of  a series  of  states  corresponding  to  sections  of  the  instruction  booklet. 
When  the  student  worked  on  a problem,  the  tutor  used  an  example  of  a correct 
solution  to  the  programming  problem  and  generated  advice  to  the  student  based 
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on  a comparison  of  the  student's  program  and  the  correct  program.  (If  there 
were  several  correct  programs  it  arbitrarily  used  a simple  and  straight- 
forward version  for  the  comparison.)  Whenever  the  student  asked  for  help 
or  ran  a program,  the  tutor  matched  the  output  of  the  student's  prugram 

against  the  output  of  the  correct  program.  If  the  results  of  the  student's 
program  were  correct,  the  tutor  congratulated  the  student  and  suggested 

starting  the  next  unit.  If  the  results  were  not  correct,  the  tutor  compared 

the  statements  in  the  two  programs  and  advised  the  student  of  the  discrepancies. 

The  Conceptual  FLOW  Tutor 

We  have  begun  development  of  a new  version  of  the  automated  tutor, 
known  as  the  Conceptual  FLOW  tutor.  The  instruction*.!  context  shown  in 
Figure  9 is  similar  to  that  of  the  earlier  automated  FLOW  tutor:  the 
student  reads  an  instruction  booklet  and  works  nr.  a computer  terminal  while 
the  tutor  monitors  the  student  and  gives  advice  if  the  student  appears  to 
be  in  trouble.  The  Conceptual  FLOW  tutor,  however,  differs  from  the  earlier 
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Figure  9.  The  FLOW 
Conceptual  Tutor 
System. 
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tutor  in  a number  of  important  ways.  1)  The  analysis  of  the  student's  work 
and  the  advice  to  the  student  can  be  handled  in  terms  of  higher  level 
programming  concepts  such  as  loops  and  order  of  execution,  while  the  original 
automated  tutor  worked  only  in  terms  of  the  FLOW  statements.  2)  The  Conceptual 
FLOW  tutor  builds  a more  complex  model  of  the  student  and  uses  that  model  to 
formulate  advice  to  the  student.  3)  The  tutor  uses  schemas  to  represent 
the  knowledge  data  base,  to  solve  problems,  and  to  interpret  student  behavior. 

In  general,  FLOW  programs  are  understood  in  a hierarchy  of  four  levels. 

At  the  highest  level  there  is  a description  of  the  function  or  output  of  the 
program.  The  second  level  describes  the  topological  structure  of  the 
program  (this  is  roughly  equivalent  to  a flow  chart  of  the  program) . The 
third  level  consists  of  the  actual  statements  in  the  program.  The  fourth 
and  final  level  is  the  list  of  keys  which  must  be  pressed  by  the  student  to 
enter  the  program. 

Typically,  the  tutor  solves  problems  by  starting  with  a description  of 
the  desired  function  and  then  determining  the  required  FLOW  statements  and 
keypresses.  The  student's  responses  are  interpreted  by  moving  up  in  the 
hierarchy  from  keypresses  or  statements  toward  the  overall  function  of  the 
program.  In  operation,  the  tutor  moves  freely  up  and  down  the  understanding 
hierarchy.  For  example,  if  the  student  has  been  given  a programming  problem 
which  requires  a sequence  of  actions  to  be  repeated,  the  tutor  realizes  that 
this  requires  a loop,  and  that  this  requires  a JUMP  statement.  The  tutor 
examines  the  student's  program  for  the  appropriate  JUMP  statement.  If  the 
statement  is  missing,  the  tutor  explains  the  need  for  a loop.  Here  the  tutor 
has  started  at  the  top  level,  dropped  down  two  levels  to  find  a fault  in  the 
student's  program,  and  finally  moved  up  one  level  to  describe  the  fault  to 


the  student. 


Schema s and  Their  Uses 


Schemas  as  used  in  the  Conceptual  FLOW  tutor  consist  of  a series  of 
slots,  each  with  a name  and  a value.  Table  2 shows  some  typical  schemas 
used  by  the  tutor.  The  first  schema  in  Table  2 is  for  the  display 
statement.  The  FUNCTION  slot  of  this  schema  has  a value  of  DISPLAY, 
indicating  that  the  function  of  the  display  statement  is  described  by  the 
higher-level  schema  for  DISPLAY.  The  STATEMENT-NUM  and  STRING  slots  are 
now  occupied  by  variables,  but  they  would  be  filled  in  for  any  particular 
instance  of  a display  statement.  The  METHOD  slot  on  this  schema  has  the 
value  PRESS-3,  an  instance  of  the  PRESS  schema.  By  examining  these  schemas 
the  tutor  can  tell  that  to  get  a display  statement,  a particular  series  of 

keys  must  be  pressed. 

The  tutor  must  have  considerable  knowledge  about  programming  in  general 
and  the  FLOW  language  in  particular.  The  tutor  uses  its  knowledge  to  direct 
its  operations.  In  the  concept-driven  mode,  the  tutor  scans  through  its 
representation  of  the  instruction  booklet  until  it  finds  the  next  problem, 
deduces  a correct  solution  for  the  problem,  and  predicts  the  next  response 
of  the  student.  If  the  student  gives  the  expected  response,  the  tutor  notes 
that  and  goes  on  to  predict  the  next  response.  An  unexpected  student  response 
puts  the  tutor  into  data-driven  mode,  where  it  uses  its  knowledge  to  interpret 
the  student's  action.  Some  actions  by  the  tutor  may  involve  both  modes  of 
operations,  as  when  the  tutor  interprets  an  unexpected  student  response  as  a 
typical  error  in  the  data-driven  mode,  then  uses  the  high  level  schema  for 
that  error  to  predict  the  next  student  response  in  the  concept-driven  mode, 
and  finally  checks  the  student  response  to  verify  its  hypothesis. 
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Table  2.  Typical  Schemas  Used  by  the  Conceptual  FLOW  Tutor 


DISPLAY-STATEMENT 
ISA  TYPEFRAME 
FUNCTION  DISPLAY 
STATEMENT-NUM  VARIABLE-2 
STRING  VARIABLE- 1 
METHOD  PRESS- 3 

VARIABLE- 2 

TYPE  VARIABLE 
CONSTRAINT  3DIGIT-NUMBER 

PRESS-3 

TYPE  PRESS 
KEY  ORDER- 1 

FUNCTION  DISPLAY-STATEMENT 

ORDER- 1 

TYPE  ORDER 
ISAINVERSE  D 
ISAINVERSE  QUOTE 
ISAINVERSE  VARIABLE- 1 
ISAINVERSE  QUOTE 
KEY-I  PRESS-3 

VARIABLE-1 

TYPE  VARIABLE 


DISPLAY-STATEMENT  is  the  name  of 
the  schema. 

DISPLAY  is  the  name  of  another  schema. 

VARIABLE-1  and  VARIABLE-2  get  filled 
in  when  the  schema  is  used. 


This  schema  indicates  that  VARIABLE-2 
must  be  a 3-digit  number. 


This  schema  expl' xns  the  METHOD  of  the 
DISPLAY-STATEMENT. 


This  schema  indicates  that  the  display 
statement  requires  the  student 
to  press  the  following  keys: 

D " . . .VARIABLE-1. 

VARIABLE-1  is  any  string 
of  characters. 
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The  Student's  Acqui  ’tion  and  Use  of  Schemas 

Just  as  a tutor  has  memory  schemas  that  can  be  used  to  direct  the  flow 
of  teaching,  students  too  have  schemas  that  guide  their  operations.  Our 
studies  of  the  processes  of  the  students  indicate  that  they  start  with 
conceptualizations  that  are  then  modified  as  they  progress  through  the 
learning  sequence.  Students  are  active  processors  of  the  information 
presented  to  them,  and  the  instructor  must  take  this  into  account.  Accordingly, 
we  must  learn  to  structure  our  learning  strategies  in  relation  to  the 
interpretations  and  prior  schemas  of  the  students.  A major  difficulty  in  the 
teaching  process  is  to  get  the  students  to  realize  their  own  misconceptions. 

Many  times  we  have  discovered  that  students  have  schemas  for  concepts,  but 
that  these  schemas  are  not  complete,  or  are  otherwise  erroneous. 

In  this  section  we  illustrate  some  of  the  stages  in  the  developing 
knowledge  schemas  of  a student  who  was  learning  FLOW.  In  this  example,  we 
were  not  using  the  automated  tutorial  program,  but  rather  were  using  a 
technique  developed  to  give  better  data  on  the  status  of  the  student's 
knowledge.  The  student  was  instructed  to  attempt  to  predict  the  behavior 
of  a program  just  prior  to  executing  it,  and  she  ras  continually  asked  to 
describe  her  thoughts,  particularly  when  her  predictions  failed  to  be  verified. 
We  pick  up  the  experiment  from  the  point  where  the  student  is  learning  to  use 
the  command  "Jump  to."  She  has  just  previously  learned  to  use  the  command 
"Print"  in  several  different  programs,  but  no  program  was  longer  than  two 
lines  and  most  contained  only  a sing'e  use  of  the  "Print"  command. 

We  start  observing  the  student  at  the  point  where  she  has  been  asked  to 
type  Program  1 onto  the  display  terminal. 


Program  1 : Q10 

020 


Print  "Rochelle" 


Jump  to  010 

Experimenter:  This  program  will  make  the  computer  repeat  the 

printing  of  the  word  "Rochelle."  What  do  you  think 
the  output  will  look  like? 

Student:  The  computer  will  print  the  word  "Rochelle"  twice. 

The  answer  is  consistent  with  the  ordinary  sense  of  the  word  "repeat."  It  is 
also  consistent  with  the  student’s  prior  experience,  for  in  previous  programs 
with  no  JUMP  statements  and  at  most  two  PRINT  statements,  any  program  that 
repeated  the  same  printout  printed  the  same  word  twice.  If  we  could 
characterize  this  student's  schema  for  the  purpose  of  the  "Jump  to"  instruction, 
it  probably  would  look  something  like  this: 

Schema  1:  If  the  instruction  is  "Jump  to  n",  then  the  computer  does 

instruction  number  n. 

Now  the  student  was  instructed  to  run  Program  1.  When  she  did  so,  the 
output  that  appeared  looked  like  this: 

Rcehel 1 eRochel 1 eRochel 1 eRochel 1 eRochel 1 eRochel 1 eRochel 1 eRochel 1 eRochel 1 eRochel 

Student:  I guess  it  keeps  repeating  until  someone  tells  it  to  stop. 

By  her  comment,  the  student  has  clearly  learned  something  more  about  the  JUMP 
statement.  To  test  what  she  had  learned,  we  asked  her  to  enter  Program  2 into 
the  computer  and  to  predict  its  outcome. 


Program  2:  010 

Print 

"Hi" 

020 

Print 

"Rochelle' 

030 

Jump 

to  010 

Experimenter:  What  dp  you  think  this  program  will  do? 
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Student:  Its  first  instruction  is  to  print  "Hi"  so  it  will  do 

"Hi",  then  it  will  (pauses)  there's  no  space,  so  it 
will  just  go  "HiRochelle"  for  the  second  instruction. 

And  then  it  will  go  back  to  the  first  instruction  which 
was  Print  "Hi",  so  it  will  just  write  "Hi"  until  we  tell 
it  to  stop. 

We  see  from  this  example  that  the  student  has  modified  Schema  1 into  a new 
form,  something  like  this: 

Schema  2:  Do  each  instruction  in  order  unless  the  instruction  is  a 

JUMP-TO.  If  the  instruction  is  JUMT-TO  n,  then  continue 
doing  instruction  n until  told  to  stop. 

Note  that  this  schema,  even  though  incorrect,  is  perfectly  consistent  with 
everything  the  student  has  seen  up  to  this  point.  She  has  derived  her 
notion  of  sequential  order  of  execution  from  earlier  programs  and  has  used 
it  here  to  predict  the  first  two  elements  of  the  output.  From  Program  1 
she  has  seen  that  the  JUMP-TO  in  that  program  caused  the  instruction  to  be 
repeated.  Hence,  she  developed  Schema  2. 

The  test  of  the  student's  schema  came  when  she  was  asked  to  run  Program  2. 
Here  is  what  happened: 

HiRochel TeHiRochel leHiRochel  1 eH  i Roche!  1 eHiRocheHeHi  RocheHeHiRocheneHi  Roche! 

Once  again  the  result  was  not  what  was  expected.  Once  again  the  schema 
for  "Jump"  had  to  be  modified. 

Student:  When  you  say  "Jump  to  the  first  instruction",  it  will 

go  to  that  and  then  I guess  it  goes  to  the  second  one 
and  if  there  isn't  a second  one  it  will  just  keep 
repeating  the  first  one.  Otherwise  it  will  repeat  both. 
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This  is  a rather  complicated  and  highly  conditionalized  notion,  but  it  is 
perfectly  consistent  with  all  example  she  las  seen.  When  she  was  asked  to 
describe  how  the  computer  actually  performed  Program  2,  she  provided  a 
correct  line-by-line  description.  Her  schema  now  might  be  characterized  as: 
Schema  3:  Do  each  instruction  in  order  unless  the  instruction  is 

"Jump  to".  If  the  instruction  is  "Jump  to  n",  then  begin 
doing  instructions  at  number  n.  If  there  are  no  more 
instructions,  stop. 

Again  we  tested  her  knowledge  by  askeng  her  to  type  Program  3 and  to 
predict  the  result: 

Program  3:  010  Jump  to  030 

020  Print  "Hi" 

030  Print  "Rochelle" 

Student:  The  computer  will  go  to  the  third  instruction  and 

print  "Rochelle",  then  to  the  second  and  print  "Hi", 
and  then  to  the  third  again  and  print  "Rochelle". 

The  actual  result  is: 

Rochelle 

Only  one  word  is  printed,  and  then  the  program  halts.  Why  did  the  student 
predict  what  she  did,  when  according  to  Schema  3 she  should  have  been  able  to 
predict  the  result  properly?  Evidently  she  has  other  schemas  about  the 
operation  of  the  computer.  Many  students  seem  to  believe  that  every  statement 
must  be  executed  at  least  once,  and  this  schema  could  apply  here.  If  so, 
this  causes  a conflict  with  Schema  3,  which  might  possibly  be  resolved  by  a 
reversion  to  one  of  the  earlier  schemas  for  JUMP-T0.  Whatever  the  reason,  it 
was  a simple  matter  for  the  student  to  modify  her  schema  for  "Jump".  When  she 
saw  that  the  output  was  the  single  word  "Rochelle'1,  she  was  readily  able  to 
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determine  why: 


W 


Student:  The  first  instruction  tells  it  to  go  on  to  the 

third  and  then  there  is  no  instruction  to  tell 
it  what  to  do  so  it  stops. 

Now,  finally,  she  seems  to  have  a complete  and  correct  schema  for  the 
JUMP-TO  instruction.  When  given  two  more  tests,  she  predicted  the  results 
correctly. 

Progxam  4:  010  Print  "Hi" 

020  Print  "Rochelle" 


030  Jump  to  020 

The  predicted  and  correct  result  is: 

H i Rochel  1 eRochel  1 eRochel  1 eRochel  1 eRochel  1 eRochel  1 eRoc; <c  '!  eRochel  1 eRochel  1 eRoch 

This  shows  that  the  student  does  not  believe  that  each  repetition  needs  to 


The  predicted  and  correct  result  is: 

H i H i H i H i H i H i H i H i H i H i H i H i H i H i H i H i H i H i H i H i H i H i H i H i H i H i H i H 1 H i H i H i H 1 H i H i H 1 H i H i H i H - 
This  shows  that  she  understands  that  not  every  line  needs  to  be  followed. 

These  examples  point  out  the  ways  by  which  a student  must  formulate 


hypotheses  about  the  concepts  which  are  being  taught,  learn  to  apply  their 
hypotheses,  and  learn  to  modify  them  when  necessary.  The  structural  frameworks 
of  learning  appear  to  be  organized  around  small,  simple  schemas  that  can  be 
applied  to  situations  wherever  deemed  appropriate.  Part  of  the  task  we  must 
face  is  to  determine  how  people  come  to  acquire,  apply,  and  modify  their  schemas 
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Our  observations  of  the  learning  process,  and  especially  our  tutorial 
studies  of  the  American  Civil  War  and  of  the  programming  lang  ;age  FLOW 
indicate  that  students  do  not  learn  new  structures  by  simply  taking  the  new 
knowledge  and  adding  it  to  what  they  have  already  acquired.  Rather,  the 
process  appears  to  be  a more  active  synthesis,  a filtering  of  the  new 
material  through  the  data  structures  already  existing,  changing  and  modifying 
both  new  and  old  material  to  make  some  structure  that  the  student  finl? 
satisfactory.  The  tutor's  goal  is  to  understand  what  the  student  is  doing, 
how  the  student  is  interpreting  the  new  information.  The  tutor  provides 
sufficient  background  material  that  the  new  materiel  can  be  properly 
interpreted.  In  addition,  since  new  knowledge  is  of  litt.'e  use  if  it  cannot 
be  retrieved  when  needed,  some  attention  must  be  paid  to  factors  that 
inf lienee  memory  retrieval.  Several  of  our  studies  were  concerned  with 
these  issues. 
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Experimental  Studies  of  the  Learning  Process 


In  an  effort  to  understand  the  processes  involved  in  complex  learning, 
we  have  studied  subjects  learning  Civil  War  history  and  hypothetical  maps. 
The  following  three  sections  summarize  our  findings  from  experiments  in 
three  separate  but  closely  related  projects. 

Civil  War  Tutorials 

In  the  examination  of  learning  through  individual  tutorials,  we  have 
engaged  in  several  long-term  tutorial  projects.  We  nave  taken  individual 
students  and  followed  the  evolving  state  of  their  knowledge  as  they  read 
and  discussed  material  about  the  Civil  War.  One  basic  text  was  used  for 
their  lessons:  The  Golden  Book  of  the  Civil  War  (1961).  This  text  was 
stripped  of  all  illustrations  and  unnecessary  chapters  w’ere  deleted.  The 
result  was  a 64-page  document  of  approximately  40,000  words.  Subjects  took 
from  5 to  11  hou  s to  read  the  text,  oo  that  reading  speed  ranged  between 
60  to  120  words  per  minute.  These  slew  rates  reflect  the  complexity  of  the 
material  and  (for  selected  subjects)  the  time  spent  examining  maps  of  the 
theater  of  war. 

One  major  exploration  concerned  the  use  of  subsidiary  material  by  the 
learners.  The  text  was  entirely  self-contained  so  that  theoretically  no 
extra  material  should  have  been  needed.  But  all  discussions  of  strategies 
and  battles  were  strictly  verbal  since  all  maps  and  illustrations  were 
removed.  Without  maps,  as  we  predicted,  good  spatial  representations  of 


the  events  could  not  be  formed. 


Compare  the  performance  of  two  subjects,  one  who  was  allowed  tree  use 
of  maps  (Subject  WM:  With  Maps),  and  one  who  was  allowed  only  a small  map 
showing  the  locations  of  all  the  states  (Subject  NM:  No  Maps).  The  differences 

are  shown  in  Table  3. 


Table  3.  Recall  of  the  Incidents  of  the  Civil  War 
after  Study  with  and  without  Maps 


With 

Maps 

No 

Ma  p s 

Overall  Accuracy 

0 

0 

4-  means 

superior  recall 

Global  Understanding 

+ 

- 

Q means 

(conceptual ization) 

satisfactory  recall 

Temporal  Accuracy 

Locally 

+ 

0 

- means 

Global ly 

- 

+ 

inferior  recall 

Detailed  Knowledge 

0 

+ 

NM  tended  to  recite  the  battles  in  temporal  order  without  regard  for 
their  locations.  WM  split  the  war  into  several  different  campaigns:  the 
eastern,  the  western  rivers,  the  southern  port  cities.  WM  tended  to  get 
local  sequences  more  accurately  than  did  NM,  and  to  understand  the  motives 
behind  the  incidents.  NM  tended  to  intermix  battles  that  had  occurred  at 
adjacent  times,  but  over  1000  miles  apart.  NM  had  a higher  accuracy  of  the 
ordering  of  events  in  general,  but  erred  in  presenting  local  sequences,  NM 
could  correct  errors  in  WM's  performance,  but  thought  of  the  entire  war  as 
one  single  compaign.  Thus,  WM  had  a better  conceptual  grasp,  but  had  poor 
knowledge  of  the  time  frame  across  areas  of  the  country.  NM  had  a better 
grasp  of  details,  but  a worse  conceptualization. 

When  subjects  learn  complex  material,  they  must  integrate  different 
sources  of  information.  For  the  Civil  War,  these  would  include: 


38 


Identities  of  the  major  characters 


The  political  situation 

The  economic  situation  and  policies  of  the  two  sides 
The  industrial  situation 

Transportation  routes  (waterways  and  railroads) 

Geography  (and  its  effect  on  transportation  routes  and  weather) 
Relative  size  and  power  of  armies  and  navies 
None  of  these  dimensions  of  information  seems  critical  to  the  understanding 
of  the  war,  but  each  adds  its  share.  Each  source  of  information  provides 
some  help  in  integrating  the  various  events,  and  adds  an  important  source 
of  organization  of  the  information.  From  our  experiences  with  these 
tutorials,  we  conclude  that  multiple  sources  of  information  are  important. 

A topic  as  complex  as  war  history  requires  the  integration  of  many  different 
aspects.  Even  when  the  relationships  are  not  direct,  subsidiary  information 
seems  useful  in  allowing  the  whole  to  form  a tighter  organizational  structure. 
Studies  on  the  Organization  and  Recall  of  Text 

In  a separate  series  of  experiments  on  how  people  learn,  organize,  and 
store  complex  bodies  of  information,  we  examined  the  recall  of  narrative 
prose  for  evidence  of  underlying  representational  structures  such  as  those 
proposed  b'  Rumelhart's  story  grammar.  (Reported  in  Gentner,  1976;  in  press.) 
According  to  the  grammar,  stories  are  decomposed  into  smaller  units,  such  as 
settings,  episodes,  actions,  reactions,  events  and  goals.  These  smaller 
units  (which  may  be  rurther  decomposed)  are  interrelated  with  causal  predicates 
Subjects  listened  to  tape  recordings  of  two  pages  from  a history  text 
describing  the  first  phase  or  battles  along  the  northern  Mississippi  area 
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(to  the  battle  of  Shiloh).  Their  developing  knowledge  state  was  followed 
by  collecting  recalls  after  each  presentation  of  the  prose  passage. 

For  analysis,  the  passage  as  divided  into  143  "facts"  which  were 
organized  according  to  two  different  structures:  1)  a serial  structure 
based  on  the  serial  order  of  the  facts  in  the  passage;  2)  a story  grammar 
structure  based  primarily  on  the  causal  relations  among  the  facts.  The 
analyses  of  the  recalls  showed  that  both  of  the  structures  are  important 
for  the  memory  of  the  passage.  We  looked  at  how  the  number  of  "neighbors" 
of  a fact  in  a given  structure  which  were  mentioned  on  Trial  n affect  the 
recall  of  that  same  fact  on  Trial  n+1.  (The  neighbors  of  a fact  are  the 
facts  adjacent  to  it  in  the  structure.  In  a serial  organization,  each  fact 
has  two  neighbors.  In  a story-grammar  organization,  each  fact  may  have  any 
number  of  neighbors  greater  than  one.) 

In  general,  as  a fact  has  more  neighbors  mentioned,  it  is  more  likely 
that  the  fact  will  be  remembered  correctly,  and  less  likely  that  it  will 
be  forgotten,  on  the  next  recall.  However,  there  are  important  differences 
in  the  effects  of  'he  serial  and  story-grammar  structures.  When  a fact  is 
absent  on  Trial  n,  the  number  of  neighbors  in  both  the  serial  and  story- 
grammar  structures  mentioned  on  Trial  N affects  the  recall  of  that  fact  on 
Trial  n+1,  but  the  number  of  neighbors  in  the  serial  structure  has  the 
greater  effect.  When  a fact  is  partially-correct  on  Trial  n,  again  neighbors 
in  both  structures  mentioned  on  Trial  n affect  the  recall  of  that  fact  on 
Trial  n+1,  but  now  the  number  of  story-grammar  neighbors  mentioned  has  the 
greater  effect.  Finally,  if  a fact  is  correct  on  Recall  n,  its  recall  on 
Trial  n+1  is  influenced  only  by  the  number  of  its  neighbors  in  the  story-grammar 
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structure  also  mentioned  on  Trial  n;  the  number  of  its  neighbors  in  the 
serial  structure  mentioned  on  Trial  n does  not  have  any  effect. 

This  pattern  of  results  has  a simple  explanation.  On  first  nearing  the 
tape  recording,  the  subjects  perceive  the  passage  as  a collection  of 
sentences  or  facts  strung  together  in  serial  order,  but  as  postions  of  the 
passage  begin  to  "make  sense,"  they  perceive  and  organize  the  passage  in 
a manner  closer  to  its  underlying  meaning  structure,  and  the  serial  order 
loses  its  initial  importance.  We  found  that  only  the  neighbors  in  the 
story-grammar  structure  actually  mentioned  by  the  subject  on  one  recall 
will  affect  performance  on  the  subsequent  recall.  The  structure  inherent 
in  a prose  passage  has  no  effect  unless  it  is  present  in  the  subject's 
memory  for  that  passage. 

Studies  on  Map  Learning 

Our  studies  of  the  learning  of  Civil  War  history  indicated  that  spatial 
information  played  an  important  role  in  the  conceptualization  of  the  campaign^ 
and  the  individual  battles.  Accordingly  we  became  interested  in  how  people 
learn  spatial  information,  particularly  maps,  and  how  they  integrate  spatial 
information  with  verbal  information.  The  experimental  studies  concentrated 
on  two  areas:  1)  The  process  by  which  maps  are  learned:  Is  there  a general 

strategy  which  people  use  when  they  study  a map?  2)  The  processes  of  storage 
and  retrieval  of  map  information:  Are  there  particular  organizations  which 

are  more  effective  than  others  for  the  storage  and  retrieval  of  map  information? 
(This  work  is  reported  in  detail  in  Shimron,  1975.) 

The  map  used  in  these  experiments  contains  a river,  a mountain  range, 
three  major  roads,  two  bridges,  and  ten  cities  (Figure  10).  A city  was 
located  at  each  road  intersection  and  bridge  and  the  major  roads  were  nameo. 
Thus,  the  map  contained  both  spatial  and  verbal  information. 
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In  the  first  experiment,  one  group  of  subjects  was  given  unlimited  time 
(approximately  12  minutes)  to  study  the  map,  while  another  group  of  subjects 
was  allowed  to  study  the  map  for  a limited  time  (6  minutes).  On  a subsequent 
recall  test  the  two  groups  gave  a similar  performance  on  questions  involving 
local  organization  (e.g.,  Which  city  is  in  the  mountains?),  but  the 
subjects  with  unlimited  time  did  significantly  better  on  questions  involving 
global  organization  (e.g.,  Name  the  cities  on  Route  7 going  south). 

In  the  second  experiment,  the  subjects  received  the  map  information  in 
three  consecutive,  cumulative  presentations.  One  group  received  one 
dimension  of  information  (e.g.,  the  mountains  and  the  river)  in  the  first 
presentation,  an  additional  kind  of  information  in  the  second  (e.g.,  the 
highways),  and  the  rest  of  the  information  (e.g.,  city  locations)  in  the  third 
presentation.  The  second  group  received  one  section  of  the.  map  (e.g.,  the 
northern  third)  in  the  first  presentation,  an  additional  section  in  the 
second  presentation,  and  the  remaining  section  of  the  map  in  the  third 
presentat ion. 
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There  was  no  difference  in  the  performance  of  the  two  groups  on  most 


questions  in  the  recall  test,  but  the  subjects  who  received  the  map  in 
sections  did  significantly  better  on  some  of  the  questions  involving  global 
* organization.  The  results  indicate  that  while  identifying  different 

dimensions  of  information  may  be  an  important  strategy  in  learning  majs, 
optimal  associations  between  these  dimensions  are  net  formed  unless  they 
are  presented  simultaneously. 

The  third  experiment  investigated  other  activities  which  could 
facilitate  the  learning  of  maps.  Two  groups  of  subjects  were  given  a map 
to  study  for  about  3 minutes.  In  addition,  one  group  was  asked  to  copy 
the  map  on  a blank  sheet  of  paper,  while  the  second  grou^.  was  asked  to  read 
two  stories  which  had  the  map  as  the  background  of  the  events  described. 

The  timr.  for  the  two  tasks  was  equal  (about  6 minutes)  . On  the  questions 
in  the  subsequent  recall  test  where  there  were  significant  differences, 
the  subjects  who  read  the  stories  did  better  than  those  who  copied  the  ma  i. 

Map  information  seems  to  be  organized  in  several  ways:  1)  by  the  general 
map  concepts  or  schemes,  such  as  highways,  rivers  and  mountains;  b)  by 
identifying  familiar  shapes  and  associating  locations  with  it;  c)  by 
locating  shapes  and  schemes  within  the  coordinates  of  the  map;  d)  by 
encoding  relations  which  specify  how  the  schematic  organizations  are 
correlated. 

* The  results  of  the  experiments  suggest  the  following  principles  of 

learning:  1)  Learning  a map  is  a gradual  process  whereby  local  connections 

t. 

between  map  elements  are  learned  first  and  overall  integration  of  the  map 
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units  is  only  achieved  later;  2)  The  learning  of  map  information  is 
facilitated  when  the  learner  can  observe  how  map  units  are  simultaneously 
organized:  By  contrast,  those  who  learn  the  map  scheme-by-scheme  (e.g., 
the  highways  separately  from  the  river  and  the  mountains)  may  have  diffi- 
culties in  relating  one  kind  of  schematic  organization  to  another; 

3)  Propositional  networks,  such  as  stories  which  are  relevant  to  the  map, 
help  to  organize  and  to  retain  map  ii  formation  in  memory. 


Summary 


Research  under  this  contract  period  has  centered  around  studies  of  the 
representation  of  knowledge  in  memory  and  studies  of  the  learning  and 
teaching  processes.  The  active  structural  network,  a type  of  semantic 
network,  has  been  used  to  represent  both  knowledge  and  procedures  within 
the  same  data  base.  Information  in  the  active  structural  network  is 
represented  as  propositions  in  the  form  of  concepts  interconnected  by 
directed  relations.  This  approach  works  well  for  local  knowledge,  but 
does  not  capture  some  of  the  higher  level  structure  which  can  be  found  in 
narratives  such  as  American  Civil  War  history.  The  event  schema  representa- 
tion has  been  developed  to  overcome  this  problem.  It  deals  directly  with 
the  way  that  people  orgaiiize  complex  events.  Event  schemas  have  been 
used  to  structure  Civil  War  tutorials  and  to  study  the  acauisition  of 
concepts  and  error  reccvery  as  students  learn  computer  programming.  Schemas 
also  are  being  applied  to  guide  the  automated  FLOW  tutor  to  store  knowledge, 
to  solve  problems  and  to  interpret  student  behavior. 

The  studies  of  the  learning  and  teaching  processes  have  been  primarily 
concerned  with  developing  models  of  the  student  and  with  the  building  of 
automated  tutorial  systems.  The  automated  tutorial  systems  have  provided  a 
controlled  environment  in  which  to  observe  learning  and  tching.  Two 
contrasting  topic  matters  have  been  used  in  the  tutorial  systems,  Civil  War 
history  and  computer  programming.  Ine  Civil  War  tutorials  convey  a large 
body  of  factual  information,  while  the  FLOW  programming  tutorials  are 
concerned  with  procedural  and  problem  solving  skills. 

The  experimental  procedure  of  "incremental  simulation"  in  which  the 
computer  performs  whatever  aspects  of  the  tutorial  project  can  be  automated 
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and  human  tutors  play  the  role  of  other  components  of  the  system,  has  proven 
to  be  an  etfectiv  method  of  proceeding.  This  technique  permits  an  emphasis 
on  the  understanding  of  the  interactions  among  components  of  the  overall 
system.  This  is  important  because  it  allows  ideas  to  be  tested  without 
having  to  be  concerned  about  all  the  details  of  the  system,  and  without 
exerting  the  time  and  effort  required  for  full  simulation.  Changes  ir 

procedure  are  possible  sooner  than  would  be  possible  were  full  simulation 
always  required. 

Our  studies  indicate  some  simple  rules  a teacher  might  follow  to 
present  material  to  a student.  Basically,  one  wants  to  provide  the 
learner  with  some  sort  of  structure  that  will  h ilp  organize  what  is  to  be 
acquired.  In  addition,  one  wants  to  provide  multiple  organizations  and 
multiple  viewpoints.  This  allows  for  redundant  coding,  causing  an  increased 
likelihood  of  later  memory  retrieval.  It  also  allows  the  information  that 
is  sought  to  be  derived  from  the  constraints  imposed  by  all  that  has  been 
recalled  up  to  a certain  point.  Even  if  these  structures  are  partially 
erroneous  and  highly  oversimplified,  they  can  be  useful  learning  tools. 
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Dr.  Stanley  L.  Cohen 
U.S.  Army  Research  Institute  for  the 
Behavioral  and  Social  Sciences 
1300  Wilson  Boulevard 
Arlington,  V,  22209 

Or.  Ralph  Ousek 

U.S.  Army  Research  Institute  for  the 
Behavioral  and  Social  Sciences 
1300  Wilson  Boulevard 
Arlington,  VA  22209 

Or.  Leon  H.  Nawrocki 
U.S.  Army  Research  Institute  for  the 
Behavioral  and  Social  Sciences 
1300  Wilson  Boulevard 
Arlington,  VA  22209 

Or.  Joseph  Ward 

U.S.  Army  Research  Institute  for  the 
Behavioral  and  Social  Sciences 
1300  Wilson  Boulevar,. 

Arlington,  VA  22209 

HQ  USAREUR  6 7th  Army 
ODCSOPS 

USAREUR  Director  of  GED 
APO  New  York  09403 

ARI  Field  Unit  - Leavenworth 
Post  Office  Box  3122 
Fort  Leavenworth,  KS  66027 


Marine  Corps 

1 Oirector,  Office  of  Manpower 
Utilization 

Headquarters,  Marine  Corps 

(Code  MPU) 

MCB  (Building  2009) 

Quantlco,  VA  22134 

1 Dr.  A.  L.  Slafkosky 

Scientific  Advisor  (Code  RD-1) 
Headquarters,  U.S.  Marine  Corps 
Washington,  DC  203B0 

1 Chief,  Academic  Department 
Education  Center 
Marine  Corps  Development  and 
Education  Command 
Marine  Corps  Base 
Quantlco,  VA  22134 

1 Mr.  E.  A.  Dover 

2711  South  Veltch  Street 
Arlington,  VA  22206 


Mr.  James  Baker 

U.S.  Army  Research  Institute  for  the 
Behavioral  and  Social  Sciences 
1300  Wilson  Boulevard 
Arlington,  VA  22209 

Or.  Milton  S.  Katz,  Chief 
individual  Training  6 Performance 
Evaluation 

U.S.  Army  Research  Institute  for  the 
Behavioral  and  Social  Sciences 
1300  Wilson  Boulevard 
Arlington,  VA  22209 


roast  Guard 


1 Mr 


Joseph  J Cowan 


Chief 


psychological  Research  Branch 


(G-P-1/62) 

U S.  Coast  Guard  Headquarters 
Washington,  DC  20590 


Other  POP 

1 Military  Assistant  for  Human  Resources 
Office  of  the  Secretary  of  Defense 
Room  3D129,  Pentagon 
Washington,  DC  <0301 

1 Advanced  Research  Projects  Agency 
Administrative  Services 
1400  Wilson  8oulevard 
Arlington,  VA  22209 
ATTN:  Ardella  Holloway 

) Or.  Harold  F.  O'Neil,  Jr. 

Advanced  Research  Projects  Agency 
Human  Resources  Research  Office 
1400  Wilson  Boulevard 
Arlington,  VA  22209 

1 Dr.  Robert  Young 

Advanced  Research  Project  ^ency 
Human  Resources  Research  ; ce 
1400  Wilson  Roulevard 
Arlington,  VA  22209 

12  Defense  Documentation  Center 
Cameron  Station,  Building  5 
Alexandria,  VA  22314 
ATTN : TC 


Other  Government 


Miscellaneous  1 

1 Or.  5carv1a  8 Anderson 
Educational  Testing  5ervice 
17  Executive  Park  Drive,  N.E. 

Atlanta,  GA  30329  1 

1 Dr.  John  Annett 

Department  of  Psychology 
lhe  University  of  Warwick 
Coventry  CV47AL 

ENGLAND  1 

1 Mr.  5amuel  8all 

Educational  Testing  Service 
Princeton,  NJ  0B540  1 

1 Dr.  Gerald  V Barrett 
University  of  Akron 
Oepartrrnt  of  Psychology 
Akron,  OH  44325 

1 

1 Or.  Bernard  M.  Bass 
University  of  Rochester 
Graduate  School  of  Management 
Rochester  NY  14627  1 

1 Or.  Ronald  P.  Carver 
5chool  of  Educa  Ion 
University  of  Missouri -Kansas  City 
5100  Rockhlll  Road 
Kansas  City,  MD  64110 

1 Century  Research  Corporation 
4113  Lee  Highway 

Arlington,  VA  22207  1 


1 Dr.  Richard  C Atkinson 
Deputy  Director 
National  5c1ence  Foundation 
1800  G 5 tree t,  N.W. 
Washington,  DC  20rSJ 


Or  Carl  Frederiksen 
Learning  Division,  Bas,  5k  Is  Group 
National  Institute  of  Education 
1 200  19th  5treet,N.W. 

Washington,  DC  20208 


1 Or.  William  Gorham,  Olrector 

Personnel  Research  and  Development 
Certer 

U.5.  Civil  5err1ce  Commission 
1900  E Street,  N.W. 

Washington,  DC  20415 

1 Dr.  Erik  McWilliams,  Director 
Technological  Innovations  In 
Education  Group 
National  Science  Foundation 
1800 G Street,  N.W.,  Room  W 650 
Washington,  DC  20550 


1 Dr.  Andrew  R.  Molnar 

Technological  Innovations  In 
Education  Group 
National  Science  Foundation 
1800  G Street,  N.W. 
Washington,  OC  20550 


1 Dr.  Marshall  5.  5m1th 
Assistant  Acting  Olrector 
Program  on  Essential  skills 
Na‘1onal  Institute  of  Education 
8rown  Building,  Room  815 
1 °th  and  M Streets,  N.W. 

Wi  shlngton,  DC  20208 


1 Dr.  Vern  Orry 

Personnel  Research  and  Development 
Center 

U.5.  Civil  5ervice  Commission 
1900  E Street,  N.W 
Washington,  DC  20415 


1 Or.  A.  Charnel 
BEB  512 

University  of  Texas 
Austin,  TX  7B712 

1 Or.  Kenneth  E.  Clark 
University  of  Rochester 
College  of  Arts  and  Sciences 
River  Campus  SUtlon 
Rochester,  NY  14627 

1 Dr.  Allan  M.  Collins 

8olt  8eranek  and  Newman,  Inc. 

50  Moulton  5treet 
Cambridge,  MA  02138 

1 Dr.  John  J.  Collins 
Vice  President 
Essex  Corporation 
6305  Camlnlto  Estrellaao 
5an  Olego,  CA  92120 

1 Dr.  Rene'  V.  Dawls 
University  of  Minnesota 
Department  of  Psychology 
Minneapolis,  MN  55455 

1 Dr.  Ruth  Oay 
Yale  University 
Department  of  Psychology 
2 Hlllhouse  Avenue 
New  Haven,  CT  06520 

1 ERIC 

Processing  and  Reference  Facility 
4B33  Rugby  Avenue 
Bethesda,  MO  20014 

1 Dr.  Barry  M Felnberg 

Bureau  of  Social  Science  Research, 
1990  M Street,  N.W. 

Washington,  DC  20036 

1 Or.  Victor  Fields 
Montgomery  College 
Department  of  Psychology 
Rockville,  MD  20850 

1 Or.  Edwin  A.  Fleishman 
Visiting  Professor 
University  of  California 
Graduate  School  of  Administration 
Irvine.  C.‘.  n’SA4 


1 


1 


1 


1 


1 


1 


1 

Inc. 


1 


1 


Dr.  Henry  J.  Has'-  ;rger  1 

University  of  California 
School  of  5oc,al  Si .ences 
Irvine,  CA  9266' 

Or.  II  0.  Havron 

Human  Sciences  Research,  Inc. 

771C  Old  5pr1ng  House  Road 

West  Gate  Industrial  Park 

McLean,  VA  22101  ^ 

HumRRO  Central  Division 

400  Plaza  Building 

Pace  Boulevard  at  Fairfield  Drive 

Pensacola,  FL  32505  1 

KumRRO/Western  Division 
27857  Berwick  Drive 
Carmel,  CA  93921 
ATTN:  Library 

HumRRO  Central  Division/Columbus  Offlca 
Suite  23,  2601  Cross  Country  Drive 
Columbus,  GA  31906  i 

HumRRO/Western  Division 
27B57  Berwick  Drive 

Carmel,  CA  93921  , 

ATTN:  Dr.  Robert  Vlneberg 

HumRRO 

Joseph  A.  Austin  Building 

1939  Goldsmith  Lane  1 

louisvllle.  KY  40218 

Dr.  Lawrence  B Johnson 
Lawrence  Juhn'on  J Associates,  Inc. 

2001  5 Street.  N.W.,  Suite  502  1 

Washington,  OC  20009 

Or.  Arnold  F.  Kanarlck 
Honeywell,  Inc. 

7600  Ridge  Parkway 
Minneapolis,  MN  55413 

Dr.  Roger  A.  Kaufman 
203  Dodd  Hall 

Florida  State  Cuiverslty  ' 

Tallahassee , l L 32306 

Dr.  5teven  W Keele 
University  oi  Oregon 

Departmen  of  Psychology  1 

Eugene,  OR  9’403 

Or.  Oavid  Klahr 

Car>  egle-Mellon  University 

Department  of  Psychology 

Pittsburgh,  PA  15213  1 

Or.  Fzra  5.  Krendel 

University  of  Pennsylvania 

Wharton  School , DH/CC 

Philadelphia,  PA  19174  ^ 

Dr.  Alma  F Lantz 

University  of  Denver 

Denver  Research  Institute 

Industrial  Economics  Division 

Denver,  CO  80210  1 

Mr.  Brian  McNally 
Educational  Testing  Service 
Princeton,  NJ  DB540 

Or.  Robert  P.  Mackle  1 

Human  Factors  Research,  Inc. 

6780  Corton  Drive 

Santa  Barbara  Research  Park 

Goleta,  CA  93017 

Dr.  William  C.  Mann 
University  of  Southern  California 
Information  Sciences  Institute 
4676  Admiralty  Way 
Marina  Oel  Rey,  CA  90291 

Dr.  Leo  Munday,  Vice  Fresldent  1 

American  College  Testing  Program 
P.0.  8ox  16B 
Iowa  City,  IA  52240 


Mr  Luigi  Petrullo 
2431  North  Edgewood  5treer 
Arlington,  VA  22207 

Dr.  Steven  M.  Pine 
University  of  Minnesota 
Department  of  Psychology 
Minneapolis,  MN  55455 

Dr  Diane  M Ramsey-Klet 
R-K  Research  S System  Design 
3947  Rldgpmont  Orive 
Malibu,  CA  90265 

Dr.  Joseph  w R'gney 

Unlvers  ty  of  Southern  Callfornn 

Behavioral  Technology  laboratories 

3717  South  irand 

Los  Angeles,  CA  90007 


Dr.  Leona  "d  L.  Rosenbaum,  Cha  v mar 
Mo. tgomery  College 
Departmert  of  Psychology 
Rockville,  MD  20850 

Or.  George  E.  Rowland 
Rowland  and  Company.  Inc. 

P.0.  8ox  61 

Haddonfleld,  NJ  08C33 

Dr.  Arthur  I.  Siegel 
Applied  Psychological  Services 
404  .ast  Lancaster  Avenue 
Wayne,  PA  15087 
Dr.  Richard  Snow 
Stanford  University 
School  of  Education 
Stanford,  CA  94305 

Ci  C.  Harold  5tone 
1428  Virginia  Avenue 
Glendale,  CA  91202 

Mr  Cs  inis  J.  Sullivan 
c/  HAISC,  8u11d1ng  119,  M.S  2 
P.0.  Box  90515 
L-s  Angeles.  fs  90009 

Dr.  Patrick  ’’uppas,  Director 
Institute  for  Mathematical  Studies 
In  the  Scc'al  Sciences 
Stanford  Unlvirslty 
Stanford,  C>  ’4306 

Dr.  K.  W L’n. zincr 
U, liver*  c.  j-'  V d.iiern  California 
Informafijn  5c  Institute 

4676  Admiral ' • 

Marina  Oel  »J291 

Dr.  Benton  e wood 

Northwestern  u,,.  iprslty 
Department  of  Psychology 
Evanst.cn,  IL  60201 

Or.  Carl  R.  Vest 
Battelle  Memorial  Institute 
Washington  Operations 
2030  M Street,  N.W. 

Washington,  DC  20036 

Or.  David  J.  Weiss 
University  of  Minnesota 
Depa-tment  of  Psychology 
N660  Elllctt  Hall 
Minneapolis,  Mil  55455 
Dr.  K.  Wescourt 
Stanford  University 
Institute  fo-  Mathematical  5tud1es 
In  the  Social  Sciences 
Stanford  CA  94305 

Dr.  Anita  'Jest 
Denver  Research  Institute 
University  of  Denver 
Denver,  CO  80210 


1 Or  Robert  Glaser,  Co-Olrector 
University  of  Pittsburgh 
3939  O'Hara  Street 
Pittsburgh,  PA  15213 


1 Mr.  A.  J.  Pesch,  President 
Eclectoch  Associates,  Inc. 
P.0.  8ox  178 

North  Stonington,  CT  0F359 


1 C .nneth  N.  Wexler 
University  of  California 
School  of  Social  Sciences 
Irvine,  CA  92o64 


